Method of Processing Moving Picture and Apparatus Thereof

ABSTRACT

A method of processing moving picture and an apparatus thereof are disclosed. A method of processing a moving picture comprising: calculating a color distribution vector for each of plural sub-frames which are generated by dividing a frame of the moving picture; generating a first order differential of the color distribution vector based on the color distribution vector; generating a second order differential of the color distribution vector based on the first order differential of the color distribution vector; and generating a feature vector of the frame based on the color distribution vector, the first order differential of the color distribution vector and the second order differential of the color distribution vector, is provided. The moving picture processing method can provide an efficient mean to determine commonality between moving pictures by extracting a feature from a frame of the moving pictures.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of commonly owned U.S. application Ser. No.12/598,199, filed on Oct. 29, 2009, which is a §371 National Stage ofInternational Application No. PCT/KR2008/003142, filed on Jun. 4, 2008;which claims the benefit of Korean Patent Application no.10-2007-0054601, filed on Jun. 4, 2007; Korean Patent Application no.10-2007-0060978, filed on Jun. 21, 2007; and Korean Patent Applicationno. 10-2008-0051688 filed on Jun. 2, 2008; all of which are herebyincorporated herein by reference.

TECHNICAL FIELD

The present invention relates to a method of processing moving pictureand an apparatus thereof.

BACKGROUND ART

According to progress of internet technology and popularization ofmoving picture shooting devices including digital cameras, UCC (usercreated contents) became a familiar terminology. A cheaper storage and afaster broadband communication network make people to share informationwith a moving picture rather than still image. Moreover services likeYouTube can provide an environment in which moving pictures can beshared easily.

But, despite of drastically increasing numbers of moving pictures sharedon internet, progress in moving picture search technology are relativelyslow. In general, moving picture search result provides a simple list ofmoving pictures which are related to a keyword inputted by a user (foran example, title of a moving picture includes the keyword).

In this case, a list which shows a same moving picture repetitively maybe provided frequently. It is required to consider commonality orsimilarity between moving pictures to improve the search result.Considering commonality and similarity between moving pictures can behelpful to detect an infringement of copyright by illegal distributionof contents.

However, for determining commonality or similarity of moving pictures, adirect comparison for moving pictures in binaries requires excessivecalculating power. Thus, a relatively small size comparison criterion(comparison data) which represents information of a moving picture isrequired.

A more efficient method of comparing moving pictures is required becauseit is a time-consuming work to compare large-size moving pictures evenby a relatively small size comparison criterion.

As one solution for a list which shows identical moving pictures orextremely similar moving pictures repetitively in a moving picturesearch result by a search engine, a clustering method of plural movingpictures by considering commonality (or similarity) is required.

SUMMARY Technical Problem

One aspect of the invention is to provide a moving picture processingmethod and an apparatus thereof to enhance efficiency of moving picturecomparison process and providing the comparison result.

One aspect of the invention is to provide a method and an apparatus togenerate a feature vector of a moving picture (frame feature vector)which can be used as a comparison criterion for determining commonalityand similarity between moving pictures.

One aspect of the invention is provide a method and an apparatus todetect a common section between two moving pictures using featurevectors of the moving pictures and to obtain time information of thecommon section.

One aspect of the invention is to provide a method and an apparatus togather moving pictures which have commonality (namely, share a commonsection) into a group.

Technical Solution

One aspect of the invention provides a method of processing a movingpicture. A color distribution vector for each of a plurality ofsub-frames generated by dividing a frame of the moving picture can becalculated. A first order differential of the color distribution vectorcan be generated based on the color distribution vector. A second orderdifferential of the color distribution vector can be generated based onthe first order differential of the color distribution vector of theframe. A feature vector of the frame can be generated based on the colordistribution vector, the first order differential of the colordistribution vector and the second order differential of the colordistribution vector.

For generating of a feature vector of a frame, a plurality of componentscan be selected, as components of the frame feature vector, from a setconsisting of components of a color distribution vector of the frame, afirst order differential of the color distribution vector and a secondorder differential of the color distribution vector.

For generating a color distribution vector of a sub-frame, an averagecolor vector of the sub-frame can be calculated by averaging colorvectors of pixels of the sub-frame.

The average color vector of the sub-frame can be normalized. In thenormalizing of the average color vector can utilize at least oneselected from the group consisting of a minimum vector and an averagevector of average color vectors of the plurality of sub-frames belongingto a time section including the sub-frame and corresponding to a samedisplay area as the sub-frame in the moving picture.

Another aspect of the invention provides a method of processing a movingpicture. A frame feature vector for each of a first moving picture and asecond moving picture can be generated. A common section of the firstmoving picture and the second moving picture can be detected bycomparing the frame feature vector of the first moving picture with theframe feature vector of the second moving picture.

For detecting of the common section, comparing of moving picturesegments can be performed. For the comparing of moving picture segments,a commonality evaluation value of a first moving picture segment and asecond moving picture segment can be generated by comparing featurevectors of p (p is an arbitrary natural number, p≧1) frames of the firstmoving picture segment of the first moving picture with feature vectorsof p frames of the second moving picture segment of the second movingpicture respectively. The p frames of the second moving picture segmentcan have a corresponding arrangement to an arrangement of the p framesof the first moving picture segment.

The commonality evaluation value can be generated based on a distancebetween the feature vector of the first moving picture segment and thecorresponding feature vector of the second moving picture segment. (Thefirst and the second moving picture segment can have a same length ofΔt.)

The comparing of moving picture segments can be is performedrepetitively for the first moving picture segment start time t1 (aftert1 from the start point of the first moving picture) and the secondmoving picture segment start time t2 (after t2 from the start point ofthe second moving picture). The t1 can be equal to or larger than 0 andbe smaller than length of the first moving picture. The t2 can be equalto or larger than 0 and be smaller than length of the second movingpicture.

In case the commonality evaluation value indicates commonality in thefirst moving picture segment and the second moving picture segment, astart time and an end time of a common section in the first movingpicture and the second moving picture can be detected respectively, bycomparing the feature vector of the first moving picture with thefeature vector of the second moving picture applying a greater fps(frames per second) than that of p frames in the first moving picturesegment and the second moving picture segment.

The comparing of moving picture segments performed by increasing thestart time t2 of the second moving picture segment from start point ofthe second moving picture (t2=0), with the start time t1 set to startpoint of the first moving picture (t1=0). In this case, for a start timetg of the second moving picture segment which make the commonalityevaluation value to show commonality in the first moving picture segmentand the second moving picture segment, the common section start-endpoint detection can compare the feature vector of the first movingpicture only with the feature vector of frame of the second movingpicture which locates after the tg from the start point.

For repeating the comparing of moving picture segments, in case thecommonality evaluation value does not satisfy a reference value that isrequired to show commonality in the first moving picture segment and thesecond moving picture segment, a change of the first moving picturesegment start time t1 and/or a change of the second moving picturesegment start time t2 can be proportional to difference between thereference value and the commonality evaluation value.

Another aspect of the invention provides a method of processing a movingpicture. A feature vector for each of a first moving picture and asecond moving picture from a moving picture set including plural movingpictures can be generated. A common section of the two moving picturescan be detected by comparing the feature vector of the first movingpicture and the feature vector of the second moving picture. In case acommon section of the first moving picture and the second moving pictureis detected, a cluster identifier matching can be performed for thefirst moving picture and the second moving picture to have a samecluster identifier. In case an old cluster identifier of a movingpicture is replaced with a new cluster identifier during the clusteridentifier matching, replacement of a cluster identifier for anothermoving picture which has the old cluster identifier (neither of thefirst and the second moving picture) to replace the old clusteridentifier with the new cluster identifier, can be performed.

A text token for moving pictures in a moving picture set can begenerated. Detection of common section and manipulation of clusteridentifier for moving pictures which share a text token can precedethose processes for moving pictures which does not share the text token.

The moving picture processing methods described above (frame featurevector generating method, common section detecting method and movingpicture clustering method) can be executed by a computer. The movingpicture processing methods can be stored, as a program, on a computerreadable medium.

Another aspect of the invention provides an apparatus of method ofprocessing a moving picture. A feature vector generating part of theapparatus can generate a feature vector each of the first moving pictureand the second moving picture. A common section detecting part of theapparatus can detect a common section of the first moving picture andthe second moving picture by comparing the feature vector of the firstmoving picture with the feature vector of the second moving picture.

A feature vector generating part can calculate color distribution vectorfor each of plural sub-frames which are generated by dividing a frame ofa moving picture. The feature vector generating part can generate afeature vector of the frame based on the color distribution vector, afirst order differential of the color distribution vector and a secondorder differential of the color distribution vector. The feature vectorgenerating part can select plural components, as components of the framefeature vector, from a set which composed of components of the colordistribution vector, the first order differential of the colordistribution vector and the second order differential of the colordistribution vector.

A common section detecting part of a moving picture processing apparatuscan include a moving picture segment comparing part. The moving picturesegment comparing part can generate a commonality evaluation valuebetween a first moving picture segment which has a start time of t1after start point of the first moving picture and a length of Δt and asecond moving picture segment which has a start time of t2 after startpoint of the second moving picture and a length of Δt. The commonalityevaluation value can be generated by comparing feature vectors of p (pis an arbitrary natural number, p≧1) frames of the first moving picturesegment with feature vectors of p frames of the second moving picturesegment which have a corresponding arrangement to an arrangement of thep frames of the first moving picture segment respectively.

The common section detecting part can include a common section start-endpoint detecting part. The common section start-end point detecting partdetects a start point and an end point of a common section for each ofthe first moving picture and the second moving picture. The start pointand the end point can be detected by comparing the feature vector of thefirst moving picture with the feature vector of the second movingpicture applying a greater fps than that of p frames in the first movingpicture segment and the second moving picture segment, in case thecommonality evaluation value generated by the moving picture segmentcomparing part indicates commonality in the first moving picture segmentand the second moving picture segment.

A moving picture processing apparatus can include a moving pictureclustering part. In case a common section between the first movingpicture and the second moving picture is detected by a common sectiondetecting part of the moving picture processing apparatus, the movingpicture clustering part can perform a cluster identifier matching. Bythe cluster identifier matching, the first moving picture and the secondmoving picture can have a same cluster identifier. In case an oldcluster identifier is replaced with a new cluster identifier during thecluster identifier matching, the moving picture clustering part canreplace the old cluster identifier of another moving picture (neither ofthe first moving picture and the second moving picture) with the newcluster identifier.

A moving picture processing apparatus can include a text tokengenerating part. The text token generating part can generate a texttoken for each a first to a third moving picture. A feature vectorgenerating part of the moving picture processing apparatus can generatea feature vector for the third moving picture.

In case there is a text token shared between the first moving pictureand the second moving picture and there is no text token shared betweenthe first moving picture and the third moving picture. A common sectiondetecting part of a moving picture processing apparatus can performcommon section detection for the first moving picture and the secondmoving picture preliminarily to common section detection for the firstmoving picture and the third moving picture.

Additional aspects and advantages of the present invention will be setforth in part in the description which follows, and in part will beobvious from the description, or may be learned by practice of theinvention.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates a frame of a moving picture and its sub-framesaccording to one embodiment of the present invention.

FIG. 2 is a flow chart of a frame feature vector generating methodaccording to one embodiment of the present invention.

FIG. 3 illustrates a relation among a color distribution vector, a firstorder differential of the color distribution vector, and a second orderdifferential of the color distribution vector according to oneembodiment of the present invention.

FIG. 4 illustrates composing of a feature vector from a colordistribution vector, a first order differential of the colordistribution vector, and a second order differential of the colordistribution vector according to one embodiment of the presentinvention.

FIG. 5 is a flow chart of moving picture common section detecting methodaccording to one embodiment of the present invention.

FIG. 6 illustrates a moving picture segment comparison process accordingto one embodiment of the present invention.

FIG. 7 is a flow chart of common section detecting method between twomoving pictures according to one embodiment of the present invention.

FIG. 8 illustrates commonality relations between two moving picturesaccording to one embodiment of the present invention.

FIG. 9 is a flow chart of moving picture clustering method according toone embodiment of the present invention.

FIG. 10 illustrates a circumstance and a result of moving pictureclustering according to one embodiment of the present invention.

FIG. 11 illustrates an arrangement result on a common time axis formoving picture of a moving picture cluster according to one embodimentof the present invention.

FIG. 12 illustrates a data structure storing information of a movingpicture cluster according to one embodiment of the present invention.

FIG. 13 illustrates process of determining a representative movingpicture in a moving picture cluster according to one embodiment of thepresent invention.

FIG. 14 illustrates a configuration of a moving picture processingapparatus according to one embodiment of the present invention.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

Embodiments of a method of processing moving picture and an apparatusthereof according to certain aspects of the invention will be describedbelow in more detail with reference to the accompanying drawings.However description of the embodiments is not to limit the presentinvention to a certain embodiment. The description of the embodimentscan be understood to include all conversions, equivalents andalternatives. In case a detail description of a related prior art mayblur a point of the present invention, the detail description can beomitted. Also, in the description with reference to the accompanyingdrawings, those components are rendered the same reference number thatare the same or are in correspondence regardless of the figure number,and redundant explanations are omitted.

FIG. 1 illustrates a frame of a moving picture and its sub-framesaccording to one embodiment of the present invention.

A frame can be a still image which composes a moving picture. A framecan be used as a unit for editing a moving picture. In general, a movingpicture can be encoded to have 24 or 30 frames per second. A highquality moving picture can be can be encoded to have 60 frames persecond.

In an embodiment of the present invention, target frames of a movingpicture from which a feature vector is extracted for moving picturecomparison do not have to maintain an encoding fps of the movingpicture. Intervals between the target frames do not have to be constant.

The first frame (110) in FIG. 1 is the first frame of a moving picture.For a time axis of a moving picture, origin of the time axis can be thestart point of the moving picture as in FIG. 1. The first frame can be astill image at the start point of the moving picture (t=0).

The second frame (120) and the third frame (130) are two adjacentframes. Time interval between the two adjacent frames can be calculatedas a reciprocal of the fps in which the frames are defined. The frames,from which a feature vector is extracted for comparison of two movingpictures, can be defined in another feature fps independent from anencoding fps of the two moving pictures.

Referring to FIG. 1, the second frame (120) is divided by a 4×4 form andthe first sub-frame (121) is one of the 16 sub-frames of the secondframe. In this embodiment of the present invention, a feature vector ofa frame is originated from color distribution information of asub-frame. A process of generating a feature vector will be described indetail, referring a flow chart of FIG. 2.

FIG. 2 is a flow chart of a frame feature vector generating methodaccording to one embodiment of the present invention. In calculating acolor distribution vector for a sub-frame (S210), a vector representingcolor distribution of a sub-frame which is defined by dividing a frameof a moving picture.

The color distribution vector is a vector which represents colordistribution information in each sub-frame. Information in eachsub-frame can be expressed by color vectors of pixels in the sub-frame.The Information of each sub-frame can be represented by a vectorrepresenting color distribution of the sub-frame.

In this embodiment, a moving picture frame divided in n×n form can haven2 sub-frames. However, a frame does not have to be divided in n×n form.A frame can be divided in m×n form (the n and m are two differentarbitrary natural numbers).

A typical color distribution vector calculation method is to obtain anaverage vector of color vectors of pixels in a sub-frame. The colordistribution vector for a sub-frame can be expressed by followingequation.

D _(i)(t)=[R _(i)(t),G _(i)(t),B _(i)(t)]  [Equation 1]

The t is a time variable indicating location of the frame on a time axiswhich has an origin at the start point of the moving picture. The i isan index of each sub-frame of the frame (i=1, 2 . . . n2). Ri(t), Gi(t)and Bi(t) are averages of red, green, blue color component in sub-framei, respectively.

The color distribution vector is expressed in a RGB color coordinatesystems. However, another color coordinate system such as YUV and CYMKcan be used to express a color vector of a pixel in the sub-frame. Acolor distribution vector of the sub-frame can be expressed in a samecolor coordinate system as the color vector of the pixel. It is obviousthat a vector in a color coordinate system can be converted to a vectorin another color coordinate system.

Normalization of the acquired color distribution vector Di(t) can beperformed additionally (S220). For this, the Di(t) can be divided byaverage of color distribution vectors which belong to a certain timesection including the t (for an example, from t−ε to t+ε). A minimum ofcolor distribution vectors of the time section can be subtracted fromthe Di(t).

According to one embodiment of the present invention, normalizing of thecolor distribution vector using a minimum and/or an average of colordistribution vectors of sub-frames corresponding to a same display areain the moving picture is described. However, normalizing method of acolor distribution vector is not limited to the described.

In generating a first order differential of the color distributionvector (S230), the first order differential of the color distributionvector can be defined as a difference between a color distributionvector of a sub-frame and a color distribution vector of anothersub-frame.

However, a first order differential may not be a vector of a samedimension with a color distribution vector. It can be a scalar valuecalculated as a difference between a component of a color distributionvector and a corresponding component of another color distributionvector. This discussion can be applied to a second order differentialalso.

The first order differential of the color distribution vector Eij(t) canbe calculated by a following mathematical equation. In this case Eij(t)is a differential vector.

E _(ij)(t)=D _(i)(t)−D _(i)(t)  [Equation 2]

The t is a time variable indicating location of the frame on a time axiswhich starts from the start point of the moving picture. The i and the jare indices of the sub-frame (i, j=1, 2 . . . n2 and n is an arbitrarynatural number). In this embodiment, the Di(t) and the Dj(t) are threedimensional vectors in RGB color coordinate system. The first orderdifferential of the color distribution vector Eij(t) can be a threedimensional vector.

In generating a second order differential of the color distributionvector (S240), the second order differential of the color distributionvector can be defined as a difference between a first order differentialand another first order differential.

As mentioned before about a first order differential, a second orderdifferential may not be a vector. The second order differential can becalculated from a difference between a first order differential andanother first order differential and the differential does not have tobe a same dimension vector with the color distribution vector or thefirst order differential.

The second order differential of the color distribution vectorA_(ijkl)(t) can be calculated by a following mathematical equation.

A _(ijkl)(t)=E _(ij)(t)−E _(kl)(t)  [Equation 3]

The t is a time variable indicating location of the frame on a time axiswhich has an origin at the start point of the moving picture. The i, j,k and l (i, j, k, l=1, 2 . . . n²) are indices of the sub-frame.Relations among the color distribution vector, the first orderdifferential of color distribution vector, the second order differentialof color distribution vector can be understood referring to FIG. 3.

In generating a feature vector of the frame (S250), a feature vector ofthe frame is generated based on the result of prior calculations (S110,S120, S130, S140).

In this embodiment, characteristics of color distribution of thesub-frame color distribution is calculated based on color vectors ofpixels in the sub-frame, in RGB color coordinate system (3 dimensional).The color distribution vector of the sub-frame, a first orderdifferential of the color distribution vector and a second orderdifferential of the color distribution vector are three dimensionalvectors. The dimension of these vectors can be subordinate to thedimension of coordinate system in which the characteristics of colordistribution of the sub-frame is expressed.

The calculated color distribution vector, the first order differentialof the color distribution vector and the second order differential ofthe color distribution vector are representing information in a frame.Thus, a feature vector representing frame information can be generatedby selecting a plurality of components from these vectors.

The feature vector can be composed by selecting more than one componentfrom a set which consists of components of the color distributionvector, the first order differential of the color distribution vectorand the second order differential of the color distribution vector. Incase h (h is an arbitrary natural number) components are selected fromthese vectors the feature vector of the frame can be an h dimensionalvector. The dimension of a feature vector can be modified for accuracyand rapidity of moving picture comparison.

Composing a feature vector from the vectors can be understood referringto FIG. 4. In FIG. 4, more than one component of a color distributionvector of a sub-frame, a first order differential of the colordistribution vector and a second order differential of the colordistribution vector. It is not indispensable that at least one componentis selected from each of the three vectors (a color distribution vectorof a sub-frame, the first order differential of the color distributionvector and the second order differential of the color distributionvector). More than one type of the three can be excluded fromcomposition of a feature vector.

The selection method described above is not the only solution forgenerating a feature vector and an additional calculation process can beutilized for generating a feature vector from the color distributionvector of the sub-frame, the first order differential of the colordistribution vector and the second order differential of the colordistribution vector.

The feature vector can be used as a finger print data of the frame.Problems of comparing entire information of a frame to determinecommonality or similarity in moving pictures can be solved by utilizinga simplified feature vector described above.

A higher dimension of a feature vector may require a higher computingpower and can provide a more accurate moving picture comparison. Thus,determining a proper dimension of the feature vector may be required.

Referring to FIG. 4, a first order differential and a second orderdifferential are vectors of a same dimension as a color distributionvector. As mentioned above, the first and the second order differentialmay not be vectors. The first order differential and the second orderdifferential can be calculated based on a just required component of acolor distribution vector of a sub-frame to form a feature vectors. Inthis case, the first order differential and the second orderdifferential can be a lower dimensional vector than a color distributionvector or be a scalar value.

A moving picture data can be separated into audio data and video data. Afeature vector can be extracted for each of the data. The descriptionfor FIG. 1 to FIG. 4 is mainly about frame feature vector extractionbased on the video data.

FIG. 5 is a flow chart of moving picture common section detecting methodaccording to one embodiment of the present invention.

In generating a feature vector of a moving picture (S410), a featurevector of the frame can be extracted for each of comparison targetmoving pictures. The feature vector is an implication of information inthe frame of the moving picture. A feature vector generating methodaccording to one embodiment of the present invention is described abovereferring to FIG. 1 to FIG. 4.

A feature vector generating method for detecting a common section (S420)is not limited to the described feature vector generating (extracting)method above. The extracted information from a frame of a moving picturecan be a basis of comparison of two moving pictures in detecting acommon section (S420).

In detecting a common section of the first moving picture and the secondmoving picture (S420), a common section between two moving pictures canbe examined by comparing feature vectors of the two moving pictures.

The Detecting of a common section (S420) can include comparing of movingpicture segments (S421) to evaluate possibility of existence of a commonsection in the two moving pictures by comparing segments of the movingpictures, and detecting detail information of the common section (S422)to obtain detail information of the common section (information about astart point and an end point of the common section in each movingpicture).

In comparing of moving picture segments (S421), by comparing a movingpicture segment of the first moving picture and with a moving picturesegment of the second moving picture, commonality in the two movingpictures can be measured.

According to one embodiment of the present invention, commonalitybetween two moving picture segments can be evaluated based on comparisonof corresponding feature vectors of moving picture segments.

The corresponding two feature vectors in the first and the second movingpicture segment frame are feature vectors of the frames which have asame time interval from the start time of each moving picture segment.

Comparing of feature vectors can be fulfilled by calculating a distanceof a feature vector of the first moving picture segment and a(corresponding) feature vector of the second moving picture segment.

According to one embodiment of the present invention, the featurevector, as mentioned above referring to FIG. 2 to FIG. 4, can be an hdimensional vector composed based on a color distribution vector of theframe, a first order differential of the color distribution vector and asecond order differential of the color distribution vector. In caseF_(b)(t1) is a b-th component of a feature vector of F(t1), a frameafter t1 from the start point of the first moving picture segment, andG_(b)(t2) is a b-th component of a feature vector of G(t2), a frameafter t2 from the start point of the second moving picture segment, adistance between the two corresponding feature vectors D(t1, t2) can bedefined by a L1 norm between them. The distance can be expressed asfollowing equation.

$\begin{matrix}{{D\left( {t_{1},t_{2}} \right)} = {\sum\limits_{b = 1}^{h}{{{F_{b}\left( t_{1} \right)} - {G_{b}\left( t_{2} \right)}}}}} & \left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack\end{matrix}$

The b is an index of a feature vector and the h is dimension of featurevector.

According to one embodiment of the present invention, the distance canbe calculated for each feature vector pairs of the first and the secondmoving picture segment. In comparing of moving picture segments,commonality evaluation value between the two moving picture segments canbe calculated based on the distance between the feature vectors. Sum oraverage of the distance of the feature vector pairs can be basis of thecommonality evaluation value.

The distance between feature vectors do not have to be expressed by theL1 norm. A L2 norm and a L1 norm with a limited maximum value etc. canbe utilized to express the distance between feature vectors. Also, incase the L1 norm value does not satisfy a certain threshold value thedistance can be a set to 0 (for example, in case the L1 norm meets thethreshold, the distance can be set to 1 and otherwise the, the distancecan be set to 0).

A commonality evaluation value satisfying a pre-determined thresholdvalue can indicate that there is commonality in the two targetsegments—the first and the second moving picture segment. The thresholdcan be determined based on preliminary test using a sample movingpicture.

In case the commonality evaluation value calculated for the first movingpicture segment and the second moving picture segment does not indicatecommonality, moving picture segment comparison can be repeated for otherstart points of moving picture segments in the first moving picture andthe second moving picture.

In case, commonality evaluation value indicates a severe differencebetween the two moving picture segments, commonality between adjacentsegments may not be expected. Thus, it can be efficient to performmoving picture segment comparison for moving picture segments which havea relative greater time interval from the present comparison targetsegments.

Thus, for varying a time variable to point a next comparison targetmoving picture segment to repeat the comparison, a variation which isproportional to difference between the commonality evaluation value andthe threshold required to indicate commonality, can be applied.

In case, commonality evaluation value indicates commonality in these twomoving picture segments, detecting start-end point of a common section(S422) can be fulfilled.

A moving picture segment comparison process according to one embodimentof the present invention can be understood referring to FIG. 6. Lengthof a moving picture segment for each of the first moving picture and thesecond moving picture is Δt. With the start point of the second movingpicture segment set to the start of the second moving picture, themoving picture segment comparison can be repeated by varying the startpoint of the first moving picture segment from the start of the firstmoving picture.

Referring to FIG. 6, for a start point of the first moving picturesegment tf, commonality evaluation value indicates commonality betweenthe two comparison target segments. Because the commonality evaluationvalue is based on comparison for relatively small number of frame,detecting of start-end point of a common section (S422) can be fulfilledto detect accurate start-end point of the common section.

By performing the comparison of moving picture segments using a smallerfps prior to the detection of common section start-end point, computingpowers required to compare many moving pictures can be saved.

In detecting of a start point and an end point of a common section(S422), in case commonality evaluation value calculated in movingpicture segment comparison (S421) indicates commonality in the twomoving picture segments, the start point and the end point of the commonsection can be detected for each of the first moving picture and thesecond moving picture.

As mentioned above, for the common section start-end point detection, agreater fps (frames per second) than the fps of frames used in movingpicture segment comparison can be applied. This can enhance accuracy ofthe common section start-end point detection and save computing powerrequired for comparing of moving picture segments.

Referring to FIG. 6, because the commonality is identified when thestart time of moving picture segment in the first moving picture reachestf, the common section detection can be limited to after tf. Namely,frames which locate only after the tf in the first moving picture can becompared with the frames of the second moving picture in the commonsection start-end point detection.

The initial part of the second moving picture can correspond to themiddle part of the first moving picture as in FIG. 6 and vice versa. Thedescription above referring to FIG. 6 can be understood even if thefirst moving picture and the second moving picture are interchanged.

FIG. 7 is a flow chart of common section detecting method between twomoving pictures according to one embodiment of the present invention andFIG. 8 illustrates commonality relations between two moving picturesaccording to one embodiment of the present invention.

A segment in the second moving picture same as ((identical/similar to)the initial segment of the first moving picture (a segment including thestart point of the first moving picture) can be detected, by repeatingthe comparison from the start point of the second moving picture (S450).4 detectable types of a common section are illustrated in FIG. 8.

In FIG. 8, the two moving pictures can be entirely overlapped in (a). In(b), the upper moving picture (the first moving picture) can be includedin the lower one (the second moving picture). In (c) and (d), the twomoving pictures show a partially overlapped common section. These 4 typecommon sections cab be detected by comparing the initial part of theupper moving picture (the first moving picture) with the lower (second)moving picture.

In case a common section is detected, start-end point of the commonsection can be detected (S460). For an example, the start point of thecommon section for each of the first moving picture and the secondmoving picture, can be the start point of a moving picture segment whichshows commonality in each moving picture. In FIG. 8, the common sectionstart point of the first moving picture can be the start point of thefirst moving picture, and the common section start point of the secondmoving picture can be the start point of a first segment which showscommonality with the moving picture segment of the first moving picturebecause the moving picture segment comparison is repeated from the startto the end of the second moving picture. When, the common section startpoint is determined, by comparing frame feature vectors of frames ofeach moving picture only after the start point, the common section endpoint can be determined.

This detected information about the common section can be stored in adatabase (S470). The common section start-end point information can beinclude time information and/or frame information. Also, in case nosegment in the second moving picture corresponding to the initial(first) segment of the first moving picture is found, comparison of thetwo moving pictures can be fulfilled about a first (initial) segment ofthe second moving picture (S452). The comparison process may be similarto the description given above referring to FIG. 6. In case nocommonality is indicated in the moving picture segment comparison (S450,S452), it is considered that there is no common section between the twomoving pictures. The common section detection of moving picture segmentrelated FIG. 7 and FIG. 8 can be performed by using a commonalityevaluation value described above referring to FIG. 5.

For the detection of common section between moving pictures, a featurevector generated from audio data can be considered additionally. Toenhance accuracy of the moving picture comparison using a frame featurevector from video data and to verify the comparison, the audio featurevector can be used. For an example, in case there is a blank section(for an example, all black/blue frames in a while or missing/damagedvideo data) in a moving picture, the audio feature vector (audio fingerprint) extracted from the audio data can be utilized.

Various methods can be used to extract the audio feature vector from theaudio data. For an example, the audio feature vector can be extractedbased on at least one of MFCC (Mel Frequency Cepstral Coefficient), PLPC(Perceptual Linear Prediction Coefficient) and LPC (Linear PredictionCoefficient) of a frame of the moving picture. It is obvious that theaudio feature vector generating method does not have to be limited tothe described above. A method known to a person having ordinary skill inthe art can be applied also.

FIG. 9 is a flow chart of moving picture clustering method according toone embodiment of the present invention. Referring FIG. 9, generating afeature vector of a first moving picture and a second moving picture(S610) and detecting a common section of the two moving pictures (S620)is performed prior to cluster identifier manipulating of the movingpicture. The two processes are described above referring to FIG. 1 toFIG. 8.

However, a feature vector generating method and a common sectiondetecting method for a moving picture clustering are not limited to thedescribed above. The clustering can be accomplished based on aninformation, reasoned on another criterion, that the two moving picturesshare a common section.

In manipulating a cluster identifier of the first moving picture and/orthe second moving picture (S630), a same cluster identifier can begranted to two moving pictures which share a common section. A clusteridentifier of another moving picture (neither of the first and thesecond moving picture) can be manipulated to the same clusteridentifier.

For an example, in case the two cluster identifiers of the two movingpictures sharing a common section are different, at least one clusteridentifier of them must be modified. An old cluster identifier of amoving picture can be modified to the new cluster identifier (a samecluster identifier as the other moving picture.

In case the two comparison target moving pictures do not share a commonsection, a new cluster identifier of a new cluster can be given to amoving picture of the two moving pictures.

It may be an exhausting work to manipulating cluster identifiers ofmoving pictures by examining all possible pairs of a plurality of movingpictures. Various methods to save computing power can be utilized.

For an example, for duplicate (identical) moving pictures a same clusteridentifier can be grated. Comparison result for one of the duplicatemoving pictures can be applied to another duplicate moving picture.

Also, efficiency of moving picture comparison can be enhanced bycomparing moving pictures which have a greater possibility to be groupedinto a single cluster prior to other moving pictures. For an example,for selecting moving pictures to be compared, it can be efficient to puta higher priority to a moving picture pair which shares a commonsection.

According to one embodiment of the present invention, the moving picturefor clustering can be collected from the web. The text token of thecollected moving picture can include a title of the moving picture, atext in description for a context and a theme of the moving picture, akeyword inputted by a user to search the moving picture and a tag of ablog post including the moving picture.

FIG. 10 illustrates a circumstance and a result of moving pictureclustering according to one embodiment of the present invention. Movingpicture clustering, for a plurality of moving picture data, can includedetecting of data which have commonality by comparing the moving picturedata and grouping of them.

Referring to FIG. 10, the cluster 1 includes the moving picture C only.It may mean there is no other moving picture which has commonality withthe moving picture C. The cluster 4 includes moving picture dataB-B′-B″-B. It means that there are 4 different moving picture data whichhave commonality in the cluster 4. The moving picture B includes 2identical (duplicate) moving pictures.

The cluster L-1 includes the moving picture B and the moving picture D.This cluster can be merged with the cluster 4. In case commonality inthe moving picture B′ of the cluster 4 and the moving picture B of thecluster L-1 is detected, the two clusters can be merged. However, incase there is commonality between the picture D and the moving pictureB, and there is no commonality in the picture D compared with the movingpicture B′ and B″, mergence of the two cluster can be optional.

FIG. 11 illustrates an arrangement result on a common time axis formoving picture of a moving picture cluster according to one embodimentof the present invention. There are shown 4 moving pictures of V1, V2,V3 and V4. These 4 moving pictures are arranged considering commonsections.

Each moving picture in a cluster can have time information about a startpoint and an end point of common section (overlapped section). Themoving picture V3 has a length of 40 seconds. The moving picture V1 hasa length of 30 seconds.

Start point:end point information of a common section in the V3 can beexpressed as 30:40 on a relative time axis which starts from the startpoint of the V3. For V1 the information of the common section can beexpressed as 0:10. It is known, from the start point time information ofthe common section, that the V1 starts 30 second later than the V3. (Thecommon section of the two moving pictures starts at O second point ofthe V1 and 30 second point of the V3.) Thus, a relative arrangement oftwo moving pictures can be determined by comparing start pointinformation of a common section expressed considering each start pointof the two moving pictures.

In this case, the start point:end point information of V1 can beexpressed in a common time axis of the two moving pictures, by addingthe time difference of start point time (30 second) to the startpoint:end point information expressed in the relative time axisdescribed above (The common time axis can be a time axis having anorigin at the start point of the most precedent moving picture). Thus,the start point:end point time information of the V1 can be converted to30:60 second (in a time axis 1). Also, start point:end point informationof the common section in the V1 can be converted to 30:40 on a time axis1 in FIG. 11.

By repeating this process for moving pictures in a cluster, relativearrangement of moving pictures sharing a common section can bedetermined and time information of the moving picture data of thecluster can be expressed on a common time axis.

FIG. 12 illustrates a data structure storing information of a movingpicture cluster according to one embodiment of the present invention.Referring to FIG. 12, a moving picture list can be generated for eachcluster described by an identifier and start point/end point timeinformation of a moving picture.

For a cluster which has a cluster identifier of 1, the moving picturelist consists of 3:0/1202, 5:220/600, 7:500/700, 9:600:800 and11:700/900. The 3, 5, 7, 9 and 11 in the list are identifiers of movingpicture (moving picture data). The 0/1202, 220/600, 500/700, 600/800 and700/900 are start point/end point of each moving picture data in acluster.

Referring to FIG. 12, the cluster information can include rankinformation and cluster date information. The rank information can be atotal numbers of moving picture data in a moving picture list. Thecluster date information can be the most early time information amongthe file generation (modification) time information of moving picturesin a moving picture list. The rank information and the cluster dateinformation can be considered for determining orders of moving picturesin a cluster.

A representative moving picture can be selected for a moving picturecluster. For an example, the representative moving picture can be amoving picture which shows general contents of the moving picturecluster. Thus, if there is a long moving picture covering the entirecontents of all moving pictures of the cluster, the moving picture canbe selected as a representative moving picture of the cluster.

In case, there is not a single moving picture covering entire movingpictures, moving pictures of a moving picture set which cover contentsof the cluster with a minimum number of element moving picture can berepresentative moving pictures of the cluster.

FIG. 13 illustrates process of determining a representative movingpicture in a moving picture cluster according to one embodiment of thepresent invention. Referring to FIG. 13, a longest moving picture ismoved to the representative moving picture list (S1201). In case thereare several longest moving pictures, a moving picture which has the bestquality can be selected by considering display quality information (orbe selected arbitrarily).

Then, among the rest moving pictures, a moving picture which has alongest play-time of un-overlapped sections with the selected movingpicture data can be selected next and moved (marked) to therepresentative moving picture list. These processes can be performedrepetitively (S1202, S1203).

By repeating these processes, a representative moving picture list canbe obtained. The entire moving picture data can be composed by using themoving picture data in the representative moving picture list.

FIG. 14 illustrates a configuration of a moving picture processingapparatus according to one embodiment of the present invention.Referring to FIG. 14, a moving picture processing apparatus (700) caninclude a feature vector generating part (710), a common sectiondetecting part (720) and moving picture clustering part (730).

The feature vector generating part (710) can load a target movingpicture, observe a divided frame of the moving picture and produce afeature vector of the frame. The moving picture processing apparatus(700) extract a feature vector representing a frame from colordistribution information of a still image which can be displayed as aframe of the moving picture. In this process, the frame can be dividedinto several sub-frames.

A color distribution vector can be obtained from color vectors of pixelsof a sub-frame. By using a first order differential and a second orderdifferential of the color distribution vector, a component of thefeature vector can be calculated. Detail description about featurevector generation was given referring to FIG. 2 to FIG. 4 above.

The common section detecting part (720) identifies a common sectionshared by the target moving pictures by comparing the moving pictures.

Detecting of a common section between the moving pictures can befulfilled by comparing moving picture feature vectors. In this process,possibility of common section existence can be detected by movingpicture segment comparison. The possibility can be expressed in acommonality evaluation value by the segment comparison. Detaildescription of calculating the commonality evaluation value was givenreferring to FIG. 5 to FIG. 8 above.

Also, in case the commonality evaluation value indicates commonalitybetween two moving pictures, the common section detecting part (720) candetect a common section occupied part from each of the two movingpictures. For this, a greater fps (frames per second) can be appliedthan the moving picture segment comparison. A start point and an endpoint of a common section can be defined as a starting/ending time (orframe) of the common section in a moving picture. Detail description ofdetecting start-end point of a common section is given above referringto FIG. 5 and redundant description will be omitted.

The moving picture clustering part (730) can group moving pictures whichshare a common section, into a cluster. Clustering of moving picturescan be fulfilled by granting a same cluster identifier to movingpictures which share a common section. In this process, a new clusteridentifier, substituting an old cluster identifier of a moving picture,can be granted to another moving picture which had the old clusteridentifier.

Detail description of manipulating a cluster identifier in the movingpicture clustering was given above referring to FIG. 9. Also, it isnoted that common section detection and clustering for moving pictureswhich share a text token can be fulfilled preliminarily.

The moving picture clustering part (730) can select a representativemoving picture for a created cluster. As described above referring toFIG. 13, a typical method of choosing the representative moving pictureis choose the longest moving picture in the cluster. Also, it was notedabove that several moving pictures, covering entire video information ofmoving pictures of the moving picture cluster, can be selected asrepresentative moving pictures.

The moving picture clustering part (730) can arrange (convert) timeinformation of plural moving pictures of the created cluster. Accordingto one embodiment of the present invention, the common section detectingpart (720) can detect start-end point of the common section of themoving picture in a relative time axis which starts at the start pointof the moving picture. As mentioned above referring to FIG. 11 and FIG.12, by the moving picture clustering part (730), time information ofeach moving picture in the cluster (start-end point and overlapping timeinformation of all moving pictures) can be expressed in a common timeaxis. The origin of the common time axis can be set to a start point ofthe most precedent (most front/initial) moving picture in the cluster.

A moving picture processing method according to one embodiment of thepresent invention can be recorded on a computer readable medium as adigital code. The computer readable medium can include all types ofmedia which can be read by a computer system. For example, a ROM, a RAM,a CD-ROM, a magnetic tape, a floppy disk, optical data storage mediumcan be used.

While the above description has pointed out novel features of theinvention as applied to various embodiments, the skilled person willunderstand that various omissions, substitutions, and changes in theform and details of the device or process illustrated may be madewithout departing from the scope of the invention. Therefore, the scopeof the invention is defined by the appended claims rather than by theforegoing description. All variations coming within the meaning andrange of equivalency of the claims are embraced within their scope.

INDUSTRIAL APPLICABILITY

According to one embodiment of the present invention, a moving pictureprocessing method and an apparatus thereof to enhance efficiency ofmoving picture comparison process and providing the comparison resultare provided.

According to one embodiment of the present invention, a method and anapparatus to generate a feature vector which can be used as a comparisoncriterion for determining commonality and similarity between movingpictures are provided.

According to one embodiment of the present invention, a method and anapparatus to detect a common section between two moving pictures usingfeature vectors of the moving pictures and to obtain time information ofthe common section are provided.

According to one embodiment of the present invention, a method and anapparatus to gather moving pictures which have commonality (namely,share a common section) into a group are provided.

What is claimed is:
 1. A method of processing a moving picture on anapparatus including at least one computer processor for executingcommands that direct operations of the apparatus and a memoryoperatively coupled to the at least one processor, the memory storingcode operable when executed with the at least one processor, the methodcomprising: calculating a color distribution vector for each of aplurality of sub-frames generated by dividing a frame of the movingpicture; generating a first order differential of the color distributionvector based on the color distribution vector; generating a second orderdifferential of the color distribution vector based on the first orderdifferential of the color distribution vector; and generating a featurevector of the frame based on the color distribution vector, the firstorder differential of the color distribution vector and the second orderdifferential of the color distribution vector.
 2. The method of claim 1,wherein the generating of a frame feature vector comprises selecting aplurality of components, as components of the frame feature vector, froma set consisting of components of the color distribution vector, thefirst order differential of the color distribution vector and the secondorder differential of the color distribution vector.
 3. The method ofclaim 1, wherein the calculating of the color distribution vectorcomprises calculating an average color vector of the sub-frame byaveraging color vectors of pixels of the sub-frame.
 4. The method ofclaim 1, wherein the color distribution vector is expressed using atleast one selected from the group consisting of RGB, YUV and CYMKcoordinate systems.
 5. The method of claim 3, wherein the calculating ofthe color distribution vector comprises normalizing the average colorvector of the sub-frame, and the normalizing of the average color vectorutilizes at least one selected from the group consisting of a minimumvector and an average vector of average color vectors of the pluralityof sub-frames belonging to a time section including the sub-frame andcorresponding to a same display area as the sub-frame in the movingpicture.
 6. A method of processing a moving picture on an apparatusincluding at least one computer processor for executing commands thatdirect operations of the apparatus and a memory operatively coupled tothe at least one processor, the memory storing code operable whenexecuted with the at least one processor, the method comprising:generating a frame feature vector for each of a first moving picture anda second moving picture; and detecting a common section of the firstmoving picture and the second moving picture by comparing the framefeature vector of the first moving picture with the frame feature vectorof the second moving picture, wherein the detecting of the commonsection comprises comparing moving picture segments by generatingcommonality evaluation value of a first moving picture segment and asecond moving picture segment by comparing feature vectors of p framesof the first moving picture segment of the first moving picture withfeature vectors, corresponding arrangement to an arrangement of the pframes of the first moving picture segment, of p frames of the secondmoving picture segment of the second moving picture, p being a naturalnumber of greater than or equal to
 1. 7. The method of claim 6, whereinthe commonality evaluation value is calculated based on a distancebetween the feature vector of the first moving picture segment and thecorresponding feature vector of the second moving picture segment. 8.The method of claim 6, wherein the distance between the feature vectorof the first moving picture segment and the feature vector of the secondmoving picture segment is defined based on at least one of an L1 norm,an L2 norm, an L1 norm with a limited maximum value and an L1 norm witha threshold function, and the commonality evaluation value is generatedbased on a sum of distances of p vector pairs between the first movingpicture segment and the second moving picture segment.
 9. A method ofprocessing a moving picture on an apparatus including at least onecomputer processor for executing commands that direct operations of theapparatus and a memory operatively coupled to the at least oneprocessor, the memory storing code operable when executed with the atleast one processor, the method comprising: generating a feature vectorfor a first moving picture and a second moving picture, respectively;detecting a common section in the two moving pictures by comparing afeature vector of the first moving picture and a feature vector of thesecond moving picture; and manipulating a moving picture clusteridentifier by matching cluster identifiers of the first moving pictureand the second moving picture, such that the first moving picture andthe second moving picture have a same cluster identifier, in case, inthe detecting of common section, a common section is detected in thefirst moving picture and the second moving picture, and replacing acluster identifier, in case an old cluster identifier of any movingpicture of the two moving pictures is replaced with a new clusteridentifier during the matching of cluster identifiers, for anothermoving picture which has the old cluster identifier with the new clusteridentifier.
 10. The method of claim 9, further comprising: generating afeature vector for a third moving picture; generating a text token forthe first to third moving picture respectively; and detecting a commonsection in the first moving picture and the third moving picture bycomparing the feature vector of the first moving picture with thefeature vector of the third moving picture, wherein, in case there is atext token shared by the first moving picture and the second movingpicture and no text token shared by the first moving picture and thethird moving picture, detecting a common section in the first movingpicture and the second moving picture precedes detecting a commonsection in the first moving picture and the third moving picture. 11.The method of claim 10, wherein the text token is obtained based on atleast one of title information, file name information, tag informationand URL information of the first to third moving picture, respectively.12. The method of claim 9, further comprising, for a plurality of movingpictures in a single cluster, converting time information of theplurality of moving pictures to a time axis, the time axis beingcommonly applied for the plurality of moving pictures.
 13. The method ofclaim 9, further comprising determining a representative moving pictureamong a plurality of moving pictures in a single cluster, wherein therepresentative moving picture a longest moving picture among theplurality of moving pictures in the single cluster.
 14. A non-transitoryrecorded medium having recorded a program for executing a method of anyof claims 1 to 13, the recorded medium being readable by a computer. 15.An apparatus of processing a moving picture, comprising: a featurevector generating part generating a feature vector for each of a firstmoving picture and a second moving picture; and a common sectiondetecting part detecting a common section in the first moving pictureand the second moving picture by comparing the feature vector of thefirst moving picture with the feature vector of the second movingpicture, wherein the feature vector generating part calculating a colordistribution vector for each of a plurality of sub-frames generated bydividing a frame of the moving picture and generating a feature vectorbased on the color distribution vector, a first order differential ofthe color distribution vector and a second order differential of thecolor distribution vector.
 16. The apparatus of claim 15, wherein thefeature vector generating part selects a plurality of components, ascomponents of the frame feature vector, from a set consisting ofcomponents of the color distribution vector, the first orderdifferential of the color distribution vector and the second orderdifferential of the color distribution vector.
 17. An apparatus ofprocessing a moving picture, comprising: a feature vector generatingpart generating a feature vector for each of a first moving picture anda second moving picture; and a common section detecting part detecting acommon section in the first moving picture and the second moving pictureby comparing a feature vector of the first moving picture with a featurevector of the second moving picture; and a moving picture clusteringpart matching cluster identifiers of the first moving picture and thesecond moving picture, such that the first moving picture and the secondmoving picture have a same cluster identifier, in case, in the detectingof common section, a common section is detected in the first movingpicture and the second moving picture, and replacing a clusteridentifier, in case an old cluster identifier of any moving picture ofthe two moving pictures is replaced with a new cluster identifier duringthe matching of cluster identifiers, for another moving picture whichhas the old cluster identifier with the new cluster identifier.
 18. Theapparatus of claim 17, wherein the feature vector generating partgenerates a feature vector for a third moving picture, and furthercomprising a text token generating part which generates a text token forthe first to third moving pictures, respectively, wherein, there is atext token shared by the first moving picture and the second movingpicture and no text token shared by the first moving picture and thethird moving picture, the common section detecting part performsdetecting a common section of the first moving picture and the secondmoving picture preliminarily to detecting a common section of the firstmoving picture and the third moving picture.